The present invention relates in general to automatic software testing and, in particular, to methods and apparatus for preventing software modifications from invalidating previously passed integration tests.
Most software programs are changed from one version to the next to implement new functionality, to improve performance, and to correct defects. Very often, a software change will lead to modifications in several different subroutines.
Once a software change is made, the modification may unintentionally affect previously tested functionality. For large and complex software, this inadvertent destruction happens quite often. As a result, during integration testing of the modified software, two sets of integration test cases are exercised. A set of regression test cases are performed to determine if previously tested functionality is still in tact after the new modifications. In addition, a set of functional test cases are performed to determine if the modification produces the new desired results.
If an error is found during the integration testing, changes made to the software to fix the error may in turn unintentionally destroy other tested functionality. A large software project typically goes through several cycles of xe2x80x9cmodify and testxe2x80x9d before the number of errors found xe2x80x9cconvergesxe2x80x9d to a satisfactory low threshold. For many large projects, running the previously passed integration tests requires a substantial investment of time (e.g., weeks to months). As a project release date approaches, the development team often decides not to rerun all the previously passed tests in an effort to save time. To contain the risk of unintentionally destroying previously passed tests, they typically decide to accept only a portion of the program modifications for new functionality or bug fixes. These decisions compromise the integrity of the software product. After an extensive review of the development processes of two of the premiere software companies, Cusumano and Yoffie describe these pitfalls in their paper, xe2x80x9cSoftware Development in Internet Timexe2x80x9d which is incorporated herein by reference.
It is well known that the cost to find and fix a software error increases exponentially as the software development process progresses through coding, unit testing, subsystem integration testing, and system integration testing. The book, xe2x80x9cSoftware Engineering Economicsxe2x80x9d, by Barry Boehm and published by Prentice-Hall in 1981, which is incorporated herein by reference, gives many such examples.
A typical software project may devote one third of the available human resources and time to integration testing. In the earlier stages of development, it is typically up to the individual programmer to develop his/her own test cases and unit test environment. Because of lack of resources and time, unit test cases are typically not as extensive as integration test cases. In particular, there is no automatic tool that can help the programmer to check whether what they are working on is in conflict with previously passed tests and with the work of other programmers until integration time. In other words, software projects typically are not effective in taking advantage of the cost and time savings in finding more errors earlier in the development process.
Some earlier attempts at solving software testing problems include methods to automate test execution such as by recording and playing back graphical user interface manipulations as disclosed in U.S. Pat. No. 5,511,185 and U.S. Pat. No. 5,600,789 (both incorporated herein by reference). Others attempt to reduce the number of test cases based on how the software was modified. The article, xe2x80x9cAnalyzing Regression Test Selection Techniques,xe2x80x9d by Rothermel et al in IEEE Transaction on Software Engineering, August 1996, discusses many such methods, and U.S. Pat. No. 5,694,540 describes one such method (both incorporated herein by reference). Still other methods generate a test environment (stubs and drivers) for unit testing such as the AUT system for Cobol programs in early IBM systems as documented in the book xe2x80x9cSoftware Testing and Evaluationxe2x80x9d, by DeMillo et al., which is incorporated herein by reference. Still others consider generating test data automatically generated by analyzing source code. The article, xe2x80x9cTechniques for Automated Test Data Generation,xe2x80x9d by C. V. Ramamoorthy et al, published in the Conference Record of the Ninth Asilomar Conference on Circuits, Systems an Computers in November, 1975 described a number of these methods.
However, prior art software testing methods suffer form certain drawbacks. For example, prior art methods do not prevent new software modifications from unintentionally failing previously passed integration tests. Further, they do not take advantage of the fact that software testing is easier and less costly during unit testing than during integration testing.